{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Transformer学习笔记\n",
    "\n",
    "论文《Attention Is All You Need》提出了一个全新的完全基于Attention机制的Transformer模型，该模型不需要CNN或者RNN，性能突出，训练更快。\n",
    "\n",
    "论文地址： [Attention Is All You Need](https://arxiv.org/pdf/1706.03762.pdf)\n",
    "\n",
    "GitHub已经有人实现了Transformer模型，比如： [Kyubyong/transformer](https://github.com/Kyubyong/transformer/blob/master/modules.py)\n",
    "\n",
    "当然，Google自己也有实现，见[tensorflow/tensor2tensor](https://github.com/tensorflow/tensor2tensor)\n",
    "\n",
    "我是根据Kyubyong的实现来理解Transformer模型的。\n",
    "\n",
    "根据论文，Transformer模型的架构如下图：\n",
    "\n",
    "![transformer architecture](images/transformer.jpg)\n",
    "\n",
    "\n",
    "稍微解释一下这个结构图。整体的结构当然还是seq2seq（encoder-decoder）架构。\n",
    "\n",
    "对于Encoder：\n",
    "\n",
    "* 6个block\n",
    "* 每个block有两层，如图左半边，分别是multi-head attention和Position-wise feed forward（其实就是MLP）\n",
    "\n",
    "对于Decoder：\n",
    "\n",
    "* 6个block\n",
    "* 每个block有三层，如图右半边，分别是multi-head attention(with mask)，multi-head attention (with encoder)，和Position-wise feed forward\n",
    "\n",
    "\n",
    "\n",
    "这张图的结构很清晰，Kyubyong的实现也是完全根据这张图的结构一步一步来的。那我们逐步就理解一下代码。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add & Norm\n",
    "\n",
    "Add & Norm这是什么操作？\n",
    "回想一下，什么情况下，会使用当前层的输出加上当前层的输入？没错，就是**残差结构**！\n",
    "所以，这里的Add就是说明这里是一个残差结构！将当前层的输出加上输入即可。\n",
    "\n",
    "那么，什么是norm？\n",
    "\n",
    "https://theneuralperspective.com/2016/10/27/gradient-topics/\n",
    "\n",
    "Normalization有很多种，但是都有一个目的：将输入转化为均值为0方差为1的分布。通常来说，归一化作用在输入送入激活函数之前，因为这样可以避免输入在激活函数的饱和区。\n",
    "\n",
    "Batch Normalization是一种优化手段，但是同时带来边缘效应：减缓梯度消失。**它的主要思想就是，在每一层，对一小批数据进行归一化**。\n",
    "\n",
    "Batch Normalization很有用，但是一般不适用与循环结构。对于这种情况，我们有Layer Normalization。\n",
    "\n",
    "关于Batch Normalization和Layer Normalization的比较，有一张图：\n",
    "\n",
    "![batch_norm_and_layer_norm](images/batch_norm_and_layer_norm.jpg)\n",
    "\n",
    "Batch Normalization是一个很重要的技术，主要也是解决梯度爆炸和梯度消失的问题。从谷歌的论文，可以获得伪代码，如下图：\n",
    "![batch normalization](images/batch_norm.png)\n",
    "\n",
    "Batch Normalization适用于CNN等，Layer Normalization适用于RNN等情景。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalize\n",
    "\n",
    "我们这里用的也是Layer Normalization。\n",
    "\n",
    "代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "def normalize(inputs, epsilon=1e-8, scope=\"normalize\", reuse=None):\n",
    "    with tf.variable_scope(scope, reuse=reuse):\n",
    "        # 获取inputs的形状\n",
    "        inputs_shape = inputs.get_shape()\n",
    "        # 参数的形状，是inputs形状的最后一行\n",
    "        # 这种情况有点类似于 f= xW + b 中，把b合并到W和x中，x最后一行加上全是1的向量，W中最后一列增加一列\n",
    "        params_shape = inputs_shape[-1:]\n",
    "        \n",
    "        # 求出inputs的均值和方差，两个张量的维度和inputs保持一致\n",
    "        mean, variance = tf.nn.moments(inputs, [-1], keep_dims=True)\n",
    "        \n",
    "        beta = tf.Variable(tf.zeros(params_shape))\n",
    "        gamma = tf.Variable(tf.ones(params_shape))\n",
    "        # 使用epsilon是为了防止数值计算错误\n",
    "        normalized = (inputs - mean) / ((variance + epsilon) ** (.5))\n",
    "        \n",
    "        # 经过线性计算，得到输出output\n",
    "        outputs = gamma * normalized + beta\n",
    "    \n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当然，对于batch normalization，谷歌的tensorflow已经有实现了。beta、gamma和epsilon等参数可以使用默认值。\n",
    "代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "def normalize(inputs):\n",
    "    return tf.layers.batch_normalization(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embedding\n",
    "词嵌入阶段，这个大家应该很熟悉了。代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "def embedding(inputs, vocab_size, num_units, zero_pad=True, scale=True, scope=\"embedding\", reuse=None):\n",
    "    \"\"\"词嵌入。\n",
    "    \n",
    "    Args:\n",
    "        inputs: 输入张量\n",
    "        vocab_size: 词典数量\n",
    "        num_units: 隐藏层单元数量，也就是常说的embedding_size\n",
    "        zero_pad: 是否增加一行全0的向量\n",
    "    \"\"\"\n",
    "    \n",
    "    with tf.variable_scope(scope, reuse=reuse):\n",
    "        lookup_table = tf.get_variable(\"lookup_table\",\n",
    "                                      dtype=tf.float32,\n",
    "                                      shape=[vocab_size, num_units],\n",
    "                                      initializer=tf.contrib.layers.xavier_initializer())\n",
    "        if zero_pad:\n",
    "            # 在lookup_table的第一行加上一行全是0的向量\n",
    "            lookup_table = tf.concat((tf.zeros(shape=[1, num_units]),\n",
    "                                      lookup_table[1:,:]),0)\n",
    "        \n",
    "        output = tf.nn.embedding_lookup(lookup_table, inputs)\n",
    "        if scale:\n",
    "            output = output * (num_units ** 0.5)\n",
    "        \n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Positional Encoding\n",
    "什么是Positional encoding? 我一开始接触这个词的时候，一脸懵逼。因为传统的seq2seq好像没有这个概念，即使是带有Attention机制的seq2seq也没有这个概念。\n",
    "\n",
    "我们看懂代码，再来回答这个问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def positional_encoding(inputs, num_units, zero_pad=True, scale=True, scope=\"positional_encoding\", reuse=None):\n",
    "    \"\"\"正弦位置编码。\n",
    "    \n",
    "    Args:\n",
    "        inputs: 输入，2维张量\n",
    "        num_units: 输出维度\n",
    "        zero_pad: 是否增加一行全0的向量\n",
    "        scale: 是否乘以num_units的平方根\n",
    "    \"\"\"\n",
    "    # N就是batch_size，T就是time_steps\n",
    "    N, T = inputs.get_shape().as_list()\n",
    "    with tf.variable_scope(scope, reuse=reuse):\n",
    "        position_ind = tf.tile(tf.expand_dims(tf.range(T), 0), [N, 1])\n",
    "\n",
    "        # 根据论文的公式，得到一个二维张量\n",
    "        position_enc = np.array([\n",
    "            [pos / np.power(10000, 2.*i/num_units) for i in range(num_units)]\n",
    "            for pos in range(T)])\n",
    "\n",
    "        # 奇数列使用cosine计算，偶数列使用sin计算\n",
    "        position_enc[:, 0::2] = np.sin(position_enc[:, 0::2])  # dim 2i\n",
    "        position_enc[:, 1::2] = np.cos(position_enc[:, 1::2])  # dim 2i+1\n",
    "\n",
    "        # 转化成一个张量，通过position就可以查出对应的数值，这一点和embedding十分相似\n",
    "        lookup_table = tf.convert_to_tensor(position_enc)\n",
    "\n",
    "        if zero_pad:\n",
    "            # 在第一行加上一行全0的向量\n",
    "            lookup_table = tf.concat((tf.zeros(shape=[1, num_units]),\n",
    "                                      lookup_table[1:, :]), 0)\n",
    "        # 根据位置，查表，获取输出\n",
    "        outputs = tf.nn.embedding_lookup(lookup_table, position_ind)\n",
    "\n",
    "        if scale:\n",
    "            outputs = outputs * num_units**0.5\n",
    "\n",
    "    return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过上面的代码分析，现在知道positional encoding是啥了吗？\n",
    "\n",
    "效果上来说，就是根据输入张量inputs的维度，生成一个index矩阵，这个矩阵也就是所谓的position，然后根据这个index，从一个根据公式生成的positiong encoding表中，查找出对应的值，组成一个新的张量，返回。\n",
    "\n",
    "这个过程简直和embedding一模一样！所以我觉得positional_encoding这个名字不如改名叫做**positional_embedding**！！@哈哈\n",
    "\n",
    "那么，回到一开始的问题，为什么需要这样一个东西呢？\n",
    "\n",
    "因为Transformer模型没有使用RNN，就不能获取时序和位置信息。为了给输入增加位置信息，使用position-encoding的技术，也就是说，不同的位置的词语，它的表示应该是不一样的。\n",
    "\n",
    "一般来说，Position encoding都是代表词语的绝对位置信息，但是相对位置信息也很重要。谷歌考虑到这一点，给出了上面的正弦和余弦公式用来计算position encoding。\n",
    "\n",
    "为什么是正弦余弦函数呢：\n",
    "\n",
    "$$\\sin(\\alpha+\\beta) = \\sin\\alpha\\cos\\beta + \\cos\\alpha\\sin\\beta$$\n",
    "$$\\cos(\\alpha+\\beta) = \\cos\\alpha\\sin\\beta - \\sin\\alpha\\cos\\beta$$\n",
    "\n",
    "这表明，位置`p+k`的向量可以表示成位置`p`的线性变换，这提供了表达相对位置信息的可能。\n",
    "\n",
    "\n",
    "有个疑问：\n",
    " \n",
    "  * **batch_size也就是N，是不是应该不大于num_units?**\n",
    "  \n",
    "\n",
    "## Multi-Head Attention\n",
    "接下来就是要实现multi-head attention了。\n",
    "\n",
    "那么multi-head attention又是啥呢？\n",
    "\n",
    "《Attention Is All You Need》一文也有一个对Attention机制的描述，大致的意思是：Attention机制实际上是一个由多个Query和K-V对组成的映射函数。\n",
    "\n",
    "$$\\text{Attention}(Q,K,V)=softmax(\\frac{QK^T}{\\sqrt{d_k}})V$$\n",
    "\n",
    "论文也画了一张图来解释：\n",
    "![Scaled dot-product attention](images/scaled_dot_product_attention.jpg)\n",
    "\n",
    "\n",
    "图的左边，也就是Scaled dot-product attention，也就是我们上面的公式描述的attention。\n",
    "\n",
    "那么multi-head attention有什么不一样的？\n",
    "从右图可以很容易的看出来，首先**对于Q、K和V，进行了一次线性映射，然后通过对他们进行切分，对每一个切分的部分，并行地运用attention（scaled dot-product attention）机制**！然后将这些不同的片合并起来，进行一次线性映射，然后得到attention输出。\n",
    "\n",
    "**Multi-Head attention这种操作是不是和CNN有点类似呢？**\n",
    "\n",
    "上面有图中的$\\text{H}$就是num_head。\n",
    "\n",
    "至此，应该对Multi-Head attention有个大致印象了。\n",
    "先理解下代码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "def multihead_attention(queries, \n",
    "                        keys, \n",
    "                        num_units=None, \n",
    "                        num_heads=8, \n",
    "                        dropout_rate=0.0, \n",
    "                        is_training=True, \n",
    "                        causality=False,\n",
    "                        scope=\"multihead_attention\",\n",
    "                        reuse=None):\n",
    "    \"\"\"\n",
    "    Args:\n",
    "        causality: 控制是否mask\n",
    "    \"\"\"\n",
    "    # 上面的参数对于有点基础的人来说大部分都应该没啥问题\n",
    "    \n",
    "    with tf.variable_scope(scope, reuse=reuse):\n",
    "        if not num_units:\n",
    "            num_units = queries.get_shape().as_list[-1]\n",
    "        \n",
    "        # 线性映射，使用dense(fully connected layer)\n",
    "        Q = tf.layers.dense(queries, num_units, activation=tf.nn.relu) # shape=(N,T,C)即(batch_size, q_time_steps, ?)\n",
    "        K = tf.layers.dense(keys, num_units, activation=tf.nn.relu) # shape=(N,T_k,C)即(batch_size, k_time_steps, ?)\n",
    "        V = tf.layer.dense(keys, num_units, activation=tf.nn.relu) # shape和K相同\n",
    "        \n",
    "        # 切分，需要concat是因为split产生了一列张量\n",
    "        # 最后一个维度切分，第一个维度合并\n",
    "        Q_ = tf.concat(tf.split(Q, num_heads, axis=2), axis=0) # (h*N, T_q, C/h)，h即num_heads\n",
    "        K_ = tf.concat(tf.split(K, num_heads, axis=2), axis=0) # (h*N, T_k, C/h) \n",
    "        V_ = tf.concat(tf.split(V, num_heads, axis=2), axis=0) # (h*N, T_k, C/h)\n",
    "        \n",
    "        # 接下来应用scale dot-product attention\n",
    "        # Q和K相乘\n",
    "        outputs = tf.matmul(Q_, tf.transpose(K_, [0, 2, 1])) # (h*N, T_q, T_k)\n",
    "        # Scale\n",
    "        outputs = outputs / (K_.get_shape().as_list()[-1] ** 0.5)\n",
    "        # Key Masking\n",
    "        key_masks = tf.sign(tf.abs(tf.reduce_sum(keys, axis=-1))) # (N, T_k)\n",
    "        key_masks = tf.tile(key_masks, [num_heads, 1]) # (h*N, T_k)\n",
    "        key_masks = tf.tile(tf.expand_dims(key_masks, 1), [1, tf.shape(queries)[1], 1]) # (h*N, T_q, T_k)\n",
    "        \n",
    "        paddings = tf.ones_like(outputs)*(-2**32+1)\n",
    "        outputs = tf.where(tf.equal(key_masks, 0), paddings, outputs) # (h*N, T_q, T_k)\n",
    "  \n",
    "        # Causality = Future blinding\n",
    "        if causality:\n",
    "            diag_vals = tf.ones_like(outputs[0, :, :]) # (T_q, T_k)\n",
    "            tril = tf.contrib.linalg.LinearOperatorTriL(diag_vals).to_dense() # (T_q, T_k)\n",
    "            masks = tf.tile(tf.expand_dims(tril, 0), [tf.shape(outputs)[0], 1, 1]) # (h*N, T_q, T_k)\n",
    "   \n",
    "            paddings = tf.ones_like(masks)*(-2**32+1)\n",
    "            outputs = tf.where(tf.equal(masks, 0), paddings, outputs) # (h*N, T_q, T_k)\n",
    "  \n",
    "        # Activation\n",
    "        outputs = tf.nn.softmax(outputs) # (h*N, T_q, T_k)\n",
    "         \n",
    "        # Query Masking\n",
    "        query_masks = tf.sign(tf.abs(tf.reduce_sum(queries, axis=-1))) # (N, T_q)\n",
    "        query_masks = tf.tile(query_masks, [num_heads, 1]) # (h*N, T_q)\n",
    "        query_masks = tf.tile(tf.expand_dims(query_masks, -1), [1, 1, tf.shape(keys)[1]]) # (h*N, T_q, T_k)\n",
    "        outputs *= query_masks # broadcasting. (N, T_q, C)\n",
    "        # Dropouts\n",
    "        outputs = tf.layers.dropout(outputs, rate=dropout_rate, training=tf.convert_to_tensor(is_training))\n",
    "        # Weighted sum\n",
    "        outputs = tf.matmul(outputs, V_) # ( h*N, T_q, C/h)\n",
    "        # scale dot-product attention 结束\n",
    "        \n",
    "        # 合并multi heads结果\n",
    "        outputs = tf.concat(tf.split(outputs, num_heads, axis=0), axis=2 ) # (N, T_q, C)\n",
    "              \n",
    "        # Residual connection\n",
    "        outputs += queries\n",
    "              \n",
    "        # Normalize\n",
    "        outputs = normalize(outputs) # (N, T_q, C)\n",
    " \n",
    "    return outputs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "疑问：\n",
    "\n",
    "* 有些步骤好像在论文的图上没有表现出来？\n",
    "\n",
    "还有个很重要的问题：**如何做mask**？我在网上找到一段解释，如下图：\n",
    "\n",
    "![multihead attention mask](images/multihead_mask.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Q,K,V是什么\n",
    "\n",
    "首先我们回归下Attention模型，这个论文有一个很好的解释：**Attention机制就是一个把Q从K-V集合中进行映射得到输出的过程**。\n",
    "\n",
    "这句话怎么理解呢？想象一下，K（也就是key）是一个高维张量，与之对应的是V（也就是value）也是一个高维张量。使用Q和K做点积，那么得到的结果是什么呢？结果就是Q在K的各个方向上的夹角。也就是说Q通过和K的点积，用来选择V中最有可能的一个。\n",
    "\n",
    "我们看一下scaled dot product attention的公式更好理解：\n",
    "\n",
    "$$\\text{Attention}(Q,K,V) = softmax(\\frac{QK^T}{\\sqrt{d_k}})V$$\n",
    "\n",
    "Q和K做点积之后，作用于softmax函数，softmax函数不用多说了，它把所有输入映射到（0,1）之间，并且与之对应的每个输出和为1。然后再和V做点积，那么得到的记过就是：在K-V集合中，通过Q查找一个最有可能的结果。那么这个概率分布，就代表了在各个分量的注意力大小。这也是**Attention**一词很好的解释。\n",
    "\n",
    "根据Transformer的整体结构图，我们发现有三个Multihead Attention，每个Multihead attention中都使用Scaled dot-product attention。那么这三个部分的Attention中的Q，K，V分别是什么呢？\n",
    "\n",
    "* Encoder层，Q，K，V都是来自上一层的输出\n",
    "* Decoder层，Q，K，V都是来着上一层的输出\n",
    "* Encoder和Decoder连接处，K，V来自上一层的Encoder，Q来自Decoder本身\n",
    "\n",
    "还有个小问题，为什么QK做点积之后，需要除以一个$\\sqrt{d_k}$呢？\n",
    "我们回想一下sigmoid函数的特点，在非常负和非常正的地方，函数趋于饱和，导数很小。除以这样一个因子，可以将其从饱和区域拉回非饱和区域，也就是说这样做可以**在一定程度上防止梯度消失**。\n",
    "\n",
    "那么，我们可以实现一个dot-product attention（暂时忽略mask）。代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "def dot_product_attention(q, k, v, mode, scale=True, dropout=0.0):\n",
    "    dot = tf.matmul(q, k, transpose_b=True)\n",
    "    if scale:\n",
    "        d_k = k.get_shape().as_list()[-1]\n",
    "        dot = dot * d_k ** -0.5\n",
    "    attention = tf.nn.softmax(dot)\n",
    "    # 加入dropout\n",
    "    attention = tf.layers.dropout(attention, rate=dropout, training=mode==tf.estimator.ModeKeys.TRAIN)\n",
    "    context = tf.matmul(attention, v)\n",
    "    return attention, context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FeedForward\n",
    "剩下的feed forward层就很简单了。代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "def feedforward(inputs, \n",
    "                num_units=[2048, 512],\n",
    "                scope=\"multihead_attention\", \n",
    "                reuse=None):\n",
    "    with tf.variable_scope(scope, reuse=reuse):\n",
    "        # Inner layer\n",
    "        params = {\"inputs\": inputs, \"filters\": num_units[0], \"kernel_size\": 1,\n",
    "                  \"activation\": tf.nn.relu, \"use_bias\": True}\n",
    "        outputs = tf.layers.conv1d(**params)\n",
    "        \n",
    "        # Readout layer\n",
    "        params = {\"inputs\": outputs, \"filters\": num_units[1], \"kernel_size\": 1,\n",
    "                  \"activation\": None, \"use_bias\": True}\n",
    "        outputs = tf.layers.conv1d(**params)\n",
    "        \n",
    "        # Residual connection\n",
    "        outputs += inputs\n",
    "        \n",
    "        # Normalize\n",
    "        outputs = normalize(outputs)\n",
    "    \n",
    "    return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但是有一个有意思的地方：\n",
    "\n",
    "**为什么使用卷积**？ 因为，**Position-wise Feed Forward Networks实际上就是窗口大小为1的一维卷积**。\n",
    "这一点，在论文里也非常直白地提出来了。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 几个疑惑\n",
    "\n",
    "### Masked Multihead attention中的mask到底是什么？在哪个步骤使用？\n",
    "为什么需要mask？见上面的截图**如何做mask**。\n",
    "\n",
    "**Mask**分为两种，一种是**sequence mask**，一种是**padding mask**。\n",
    "sequence mask是为了保证decoder只注意到前面的n个词，padding mask是为了忽略输入序列的补齐位置的影响。\n",
    "\n",
    "那么sequence mask具体怎么做呢？\n",
    "哈佛大学有一个文章，有以下解释：\n",
    "\n",
    "![sequence_mask](images/sequence_mask.PNG)\n",
    "\n",
    "tensorflow里面有sequence_mask的实现。\n",
    "\n",
    "那么padding mask怎么做呢？\n",
    "就是给补齐的位置，设置一个非常大的负数。因为padding mask这一个步骤需要在softmax之前做完，给padding的位置设置大的负数的时候，经过softmax计算，它的值会趋于0，这就可以抹去padding位置的影响。\n",
    "\n",
    "GitHub也有一个实现：[attention_tf](https://github.com/bojone/attention/blob/master/attention_tf.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 不错的文章\n",
    "\n",
    "https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html\n",
    "\n",
    "http://nlp.seas.harvard.edu/2018/04/03/attention.html\n",
    "\n",
    "https://kexue.fm/archives/4765\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
